{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2408e6f9-61cf-4de6-8ff9-7cbad4ad2fd2",
   "metadata": {
    "id": "2408e6f9-61cf-4de6-8ff9-7cbad4ad2fd2"
   },
   "source": [
    "# Labeling the [Civil Comments](https://huggingface.co/datasets/civil_comments) dataset using Autolabel\n",
    "\n",
    "This dataset contains public comments collected from news websites, the task is a binary classification task -- is the provided comment toxic or not"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cc445ea9-a196-4e5e-a7ce-d3228a66ca27",
   "metadata": {
    "id": "cc445ea9-a196-4e5e-a7ce-d3228a66ca27"
   },
   "source": [
    "## Install Autolabel\n",
    "Plus, setup your OpenAI API key, since we'll be using gpt-3.5-turbo as our LLM for labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3bc51e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install 'refuel-autolabel[openai]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2bf1cc53-1180-411e-8436-16ee75350fbf",
   "metadata": {
    "id": "2bf1cc53-1180-411e-8436-16ee75350fbf"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# provide your own OpenAI API key here\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-XXXXXXXXXXXXXXXXXXXXXXXX\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c64fa7fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "logging.getLogger(\"autolabel\").setLevel(logging.ERROR)\n",
    "logging.getLogger(\"langchain\").setLevel(logging.ERROR)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "19ca0a8a-b411-4043-8e0d-7e443a6f1042",
   "metadata": {
    "id": "19ca0a8a-b411-4043-8e0d-7e443a6f1042"
   },
   "source": [
    "## Download the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "038a0c62-f6d9-48c0-9cb6-5ea3a718c5af",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "038a0c62-f6d9-48c0-9cb6-5ea3a718c5af",
    "outputId": "64a14c2b-31a5-4175-e22d-647ddf2a4b08"
   },
   "outputs": [],
   "source": [
    "from autolabel import get_data\n",
    "\n",
    "get_data(\"civil_comments\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "46c5dd71-9ac9-47ee-883f-bd87b0216149",
   "metadata": {
    "id": "46c5dd71-9ac9-47ee-883f-bd87b0216149"
   },
   "source": [
    "This downloads two datasets:\n",
    "\n",
    "* `test.csv`: This is the larger dataset we are trying to label using LLMs\n",
    "* `seed.csv`: This is a small dataset where we already have human-provided labels"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "384fea2c-64b7-4463-b4fc-ec90cbddd82b",
   "metadata": {
    "id": "384fea2c-64b7-4463-b4fc-ec90cbddd82b"
   },
   "source": [
    "## Start the labeling process!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4e2da065-0354-40de-aa0d-c56328ecc392",
   "metadata": {
    "id": "4e2da065-0354-40de-aa0d-c56328ecc392"
   },
   "source": [
    "### Experiment #1: Very simple guidelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "41f68f38-cf3e-4279-983f-7828e0c32b01",
   "metadata": {
    "id": "41f68f38-cf3e-4279-983f-7828e0c32b01"
   },
   "outputs": [],
   "source": [
    "from autolabel import LabelingAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9b311204-69f5-48e7-a00a-510e537c463f",
   "metadata": {
    "id": "9b311204-69f5-48e7-a00a-510e537c463f"
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "    },\n",
    "    # This is the LLM we will use for labeling\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"gpt-3.5-turbo-0301\",\n",
    "    },\n",
    "    # These are the instructions to the LLM\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"Does the provided comment contain 'toxic' language? Say toxic or not\",\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"example_template\": \"Input: {example}\\nOutput: {label}\",\n",
    "    },\n",
    "}\n",
    "\n",
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config=config)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8deec367-670f-459e-9e77-e051595df101",
   "metadata": {
    "id": "8deec367-670f-459e-9e77-e051595df101"
   },
   "source": [
    "Let's review the configuration file above. You'll notice the following useful keys:\n",
    "\n",
    "* `task_type`: `classification` (since it's a classification task)\n",
    "* `model`: `{'provider': 'openai', 'name': 'gpt-3.5-turbo'}` (use a specific OpenAI model)\n",
    "* `prompt.task_guidelines`: Is the provided comment 'toxic' or 'not toxic'? (how we describe the task to the LLM)\n",
    "* `prompt.labels`: ['toxic', 'not toxic'] (the two labels to choose from)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "658fa25d-b205-43f9-9953-55efaae9df46",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 310,
     "referenced_widgets": [
      "e61c6dd36edf4195a4ac600fc0faa24f",
      "902b500a2ee741d892791b8d430111b9"
     ]
    },
    "id": "658fa25d-b205-43f9-9953-55efaae9df46",
    "outputId": "af0b88bd-8080-48c6-eb66-5754dacc20b2"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b944710e2ed425a8959bb5441cbe98d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $4.3302 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0022 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$4.3302\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0022\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does the provided comment contain 'toxic' language? Say toxic or not\n",
      "\n",
      "You will return the answer with just one element: \"the correct label\"\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "471a4d8b-bfeb-4a18-8b5b-44a3d0d4ffe3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 562,
     "referenced_widgets": [
      "0828040fcf104de9935d7a157d225b16",
      "43aca6448ea54e8e9ccb8d7a77930812"
     ]
    },
    "id": "471a4d8b-bfeb-4a18-8b5b-44a3d0d4ffe3",
    "outputId": "809636d6-ed5d-4961-9042-d641585034eb"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ed2b945b55c412e9edf4b49b2176cb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/nihit/Documents/refuel-ai.nosync/virtualenv/lib/python3.8/site-packages/sklearn/metrics/_classification.py:1\n",
       "318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted\n",
       "samples. Use `zero_division` parameter to control this behavior.\n",
       "  _warn_prf(average, modifier, msg_start, len(result))\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classification_report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   not toxic       1.00      0.20      0.33         5\n",
      "       toxic       0.43      1.00      0.60         3\n",
      "\n",
      "    accuracy                           0.50         8\n",
      "   macro avg       0.71      0.60      0.47         8\n",
      "weighted avg       0.79      0.50      0.43         8\n",
      "\n",
      "Actual Cost: 0.0169\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.5      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.08            </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.5     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.08           \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d4cba7b5-bd71-42c3-98c3-7a7d3acd80f5",
   "metadata": {
    "id": "d4cba7b5-bd71-42c3-98c3-7a7d3acd80f5"
   },
   "source": [
    "### Experiment #2: Few-shot prompting to provide helpful examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7b80df85-4ec4-4bcb-a4dc-e497ff6b0876",
   "metadata": {
    "id": "7b80df85-4ec4-4bcb-a4dc-e497ff6b0876"
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "        \"delimiter\": \",\",\n",
    "    },\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"gpt-3.5-turbo-0301\",\n",
    "    },\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\":  \"Does the provided comment contain 'toxic' language? Say toxic or not.\",\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"few_shot_examples\": [\n",
    "            {\n",
    "                \"example\": \"It's ridiculous that these guys are being called 'protesters'. Being armed is a threat of violence, which makes them terrorists.\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done!\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This bitch is nuts. Who would read a book by a woman\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"It was a great show. Not a combo I'd of expected to be good together but it was.\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "        ],\n",
    "        \"few_shot_selection\": \"fixed\",\n",
    "        \"few_shot_num\": 4,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: {label}\",\n",
    "    },\n",
    "}\n",
    "\n",
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ad22c757-b1f8-419f-85dc-8bf7ea232b3f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 560,
     "referenced_widgets": [
      "8724335afccc4d8182068498544d0ed1",
      "7bdf0bcc39004bbb908b20ab07174d48"
     ]
    },
    "id": "ad22c757-b1f8-419f-85dc-8bf7ea232b3f",
    "outputId": "f0f45e89-68ee-43d7-d90d-740ba70688fe"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "26ba1bf44337429198ea91abb5199939",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $4.7051 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0024 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$4.7051\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0024\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does the provided comment contain 'toxic' language? Say toxic or not.\n",
      "\n",
      "You will return the answer with just one element: \"the correct label\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: It's ridiculous that these guys are being called 'protesters'. Being armed is a threat of violence, which makes them terrorists.\n",
      "Output: toxic\n",
      "\n",
      "Input: This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done!\n",
      "Output: not toxic\n",
      "\n",
      "Input: This bitch is nuts. Who would read a book by a woman\n",
      "Output: toxic\n",
      "\n",
      "Input: It was a great show. Not a combo I'd of expected to be good together but it was.\n",
      "Output: not toxic\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(dataset=ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b586f2ad-621c-4ea7-9d48-8cb47d058e97",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 167,
     "referenced_widgets": [
      "58b666c148a64a78b0be6c012f3598f1",
      "3adc44fde45c472e892a4a0b6a388e87"
     ]
    },
    "id": "b586f2ad-621c-4ea7-9d48-8cb47d058e97",
    "jupyter": {
     "outputs_hidden": true
    },
    "outputId": "ab1dc87f-2d2b-4fd1-83a8-c492a900f2d2",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a164eb4ddd914759b910264bafb8ca8b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classification_report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "   not toxic       0.98      0.66      0.79        87\n",
      "       toxic       0.29      0.92      0.44        13\n",
      "\n",
      "    accuracy                           0.69       100\n",
      "   macro avg       0.63      0.79      0.61       100\n",
      "weighted avg       0.89      0.69      0.74       100\n",
      "\n",
      "Actual Cost: 0.0357\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.69     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.69    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5c97a1a0-f655-4ea6-9f6d-1fd00f3479bd",
   "metadata": {
    "id": "5c97a1a0-f655-4ea6-9f6d-1fd00f3479bd"
   },
   "source": [
    "### Experiment #3: Improving prompts with error analysis (Prompt engineering)\n",
    "\n",
    "`agent.run()` returned an output Pandas Dataframe. We'll review the mistakes and update the labeling guidelines to \"teach\" the LLM our content moderation policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "69706adf-4474-46a0-ba85-2b8c2eef8392",
   "metadata": {
    "id": "69706adf-4474-46a0-ba85-2b8c2eef8392"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\"max_colwidth\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "3b6be939-53c3-4a44-acf8-1c64ed27888a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "3b6be939-53c3-4a44-acf8-1c64ed27888a",
    "outputId": "17ce0ad3-80e8-4f31-e2d5-5d8297ff5911",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>example</th>\n",
       "      <th>label</th>\n",
       "      <th>ToxicCommentClassification_label</th>\n",
       "      <th>ToxicCommentClassification_error</th>\n",
       "      <th>ToxicCommentClassification_successfully_labeled</th>\n",
       "      <th>ToxicCommentClassification_annotation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>@Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk….</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xb5\\x05\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\xf9\\x80\\x04\\x95\\xee\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c\\xc8@Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk\\xe2\\x80\\xa6.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x9f\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: @Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk\\xe2\\x80\\xa6.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI &amp; WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xdd\\x06\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\x8d\\x01\\x00\\x00\\x80\\x04\\x95\\x82\\x01\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94XY\\x01\\x00\\x00Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI &amp; WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X0\\x04\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI &amp; WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Let’s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen’s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes “it severely constrains our ability to give out property tax relief.”  So Berkowitz wants the ability to confiscate more of the citizen’s money to provide property relief. How about taking less of that money in the first place?  Let’s hope Proposition 8 passes.</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\x83\\n\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B`\\x03\\x00\\x00\\x80\\x04\\x95U\\x03\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X,\\x03\\x00\\x00Let\\xe2\\x80\\x99s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen\\xe2\\x80\\x99s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes \\xe2\\x80\\x9cit severely constrains our ability to give out property tax relief.\\xe2\\x80\\x9d  So Berkowitz wants the ability to confiscate more of the citizen\\xe2\\x80\\x99s money to provide property relief. How about taking less of that money in the first place?  Let\\xe2\\x80\\x99s hope Proposition 8 passes.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x03\\x06\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Let\\xe2\\x80\\x99s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen\\xe2\\x80\\x99s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes \\xe2\\x80\\x9cit severely constrains our ability to give out property tax relief.\\xe2\\x80\\x9d  So Berkowitz wants the ability to confiscate more of the citizen\\xe2\\x80\\x99s money to provide property relief. How about taking less of that money in the first place?  Let\\xe2\\x80\\x99s hope Proposition 8 passes.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>What would Jerry Prevo think about that story? Exactly what I think about his....</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xc7\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\x82\\x80\\x04\\x95w\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cQWhat would Jerry Prevo think about that story? Exactly what I think about his....\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X(\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: What would Jerry Prevo think about that story? Exactly what I think about his....\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>When wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95c\\x05\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\xd0\\x80\\x04\\x95\\xc5\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c\\x9fWhen wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94Xv\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: When wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>IT IS DOA! As Collins announced her intentions to vote NO!</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\x99\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94Ck\\x80\\x04\\x95`\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c:IT IS DOA! As Collins announced her intentions to vote NO!\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x11\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: IT IS DOA! As Collins announced her intentions to vote NO!\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Lets make some fact based statements. Half of them under 21???? fake news</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xb7\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94Cz\\x80\\x04\\x95o\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cILets make some fact based statements. Half of them under 21???? fake news\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X \\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Lets make some fact based statements. Half of them under 21???? fake news\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>this is *&amp;^%ing outrageous. The prosecutor should be immediately removed from his job.</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xd1\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\x87\\x80\\x04\\x95|\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cVthis is *&amp;^%ing outrageous. The prosecutor should be immediately removed from his job.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X-\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: this is *&amp;^%ing outrageous. The prosecutor should be immediately removed from his job.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95\\xa9\\x0b\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\xf3\\x03\\x00\\x00\\x80\\x04\\x95\\xe8\\x03\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X\\xbf\\x03\\x00\\x00I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x96\\x06\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Don’t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don’t need to. \\n\\nTimmy’s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald’s is moving up their automation timelines.</td>\n",
       "      <td>not toxic</td>\n",
       "      <td>toxic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>b'\\x80\\x04\\x95S\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\xc8\\x01\\x00\\x00\\x80\\x04\\x95\\xbd\\x01\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X\\x94\\x01\\x00\\x00Don\\xe2\\x80\\x99t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don\\xe2\\x80\\x99t need to. \\n\\nTimmy\\xe2\\x80\\x99s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald\\xe2\\x80\\x99s is moving up their automation timelines.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94Xk\\x04\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Don\\xe2\\x80\\x99t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don\\xe2\\x80\\x99t need to. \\n\\nTimmy\\xe2\\x80\\x99s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald\\xe2\\x80\\x99s is moving up their automation timelines.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            example  \\\n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            @Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk….   \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI & WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.   \n",
       "5                                                                                                                                                                  Let’s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen’s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes “it severely constrains our ability to give out property tax relief.”  So Berkowitz wants the ability to confiscate more of the citizen’s money to provide property relief. How about taking less of that money in the first place?  Let’s hope Proposition 8 passes.   \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 What would Jerry Prevo think about that story? Exactly what I think about his....   \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   When wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.   \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       IT IS DOA! As Collins announced her intentions to vote NO!   \n",
       "28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Lets make some fact based statements. Half of them under 21???? fake news   \n",
       "33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           this is *&^%ing outrageous. The prosecutor should be immediately removed from his job.   \n",
       "34  I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone   \n",
       "37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   Don’t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don’t need to. \\n\\nTimmy’s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald’s is moving up their automation timelines.   \n",
       "\n",
       "        label ToxicCommentClassification_label  \\\n",
       "2   not toxic                            toxic   \n",
       "4   not toxic                            toxic   \n",
       "5   not toxic                            toxic   \n",
       "8   not toxic                            toxic   \n",
       "9   not toxic                            toxic   \n",
       "15  not toxic                            toxic   \n",
       "28  not toxic                            toxic   \n",
       "33  not toxic                            toxic   \n",
       "34  not toxic                            toxic   \n",
       "37  not toxic                            toxic   \n",
       "\n",
       "   ToxicCommentClassification_error  \\\n",
       "2                              None   \n",
       "4                              None   \n",
       "5                              None   \n",
       "8                              None   \n",
       "9                              None   \n",
       "15                             None   \n",
       "28                             None   \n",
       "33                             None   \n",
       "34                             None   \n",
       "37                             None   \n",
       "\n",
       "    ToxicCommentClassification_successfully_labeled  \\\n",
       "2                                              True   \n",
       "4                                              True   \n",
       "5                                              True   \n",
       "8                                              True   \n",
       "9                                              True   \n",
       "15                                             True   \n",
       "28                                             True   \n",
       "33                                             True   \n",
       "34                                             True   \n",
       "37                                             True   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ToxicCommentClassification_annotation  \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    b'\\x80\\x04\\x95\\xb5\\x05\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\xf9\\x80\\x04\\x95\\xee\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c\\xc8@Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk\\xe2\\x80\\xa6.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x9f\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: @Rmiller101 - Spoken like a true elitist. But look out bud. The re-awakening in Europe, Brexit and now the Trump victory are just the beginning. The unwashed masses are coming for you and your ilk\\xe2\\x80\\xa6.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     b'\\x80\\x04\\x95\\xdd\\x06\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\x8d\\x01\\x00\\x00\\x80\\x04\\x95\\x82\\x01\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94XY\\x01\\x00\\x00Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI & WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X0\\x04\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Sorry you missed high school. Eisenhower sent troops to Vietnam after the French withdrew in 1954 and before that America was providing 50% of the cost of that war. WWI & WWII were won by Democrat Presidents and the last win a Republican had was 1865  but the did surrender in Korea and Vietnam and fail to win in desert Storm or these two wars.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "5                                                                                                                                                                                                        b'\\x80\\x04\\x95\\x83\\n\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B`\\x03\\x00\\x00\\x80\\x04\\x95U\\x03\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X,\\x03\\x00\\x00Let\\xe2\\x80\\x99s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen\\xe2\\x80\\x99s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes \\xe2\\x80\\x9cit severely constrains our ability to give out property tax relief.\\xe2\\x80\\x9d  So Berkowitz wants the ability to confiscate more of the citizen\\xe2\\x80\\x99s money to provide property relief. How about taking less of that money in the first place?  Let\\xe2\\x80\\x99s hope Proposition 8 passes.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x03\\x06\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Let\\xe2\\x80\\x99s see if I understand this; Berkowitz announces a $14M surplus then he rails against Proposition 8 because it would constrain his ability to confiscate even more money from citizens. Berkowitz and his allies on the Assembly changed the tax cap language to facilitate confiscating greater amounts of citizen\\xe2\\x80\\x99s money. Now why would a mayor who professes fidelity with the taxpayers change the law to enable him to take more money from them? Probably because he plans to take more money from them.  Berkowitz said if Proposition 8 passes \\xe2\\x80\\x9cit severely constrains our ability to give out property tax relief.\\xe2\\x80\\x9d  So Berkowitz wants the ability to confiscate more of the citizen\\xe2\\x80\\x99s money to provide property relief. How about taking less of that money in the first place?  Let\\xe2\\x80\\x99s hope Proposition 8 passes.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             b'\\x80\\x04\\x95\\xc7\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\x82\\x80\\x04\\x95w\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cQWhat would Jerry Prevo think about that story? Exactly what I think about his....\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X(\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: What would Jerry Prevo think about that story? Exactly what I think about his....\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "9                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              b'\\x80\\x04\\x95c\\x05\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\xd0\\x80\\x04\\x95\\xc5\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c\\x9fWhen wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94Xv\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: When wil the  indigenous  be accountable and take responsibility instead of blaming others ?  They have to be accountable and pay their own way or go it alone.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "15                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          b'\\x80\\x04\\x95\\x99\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94Ck\\x80\\x04\\x95`\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8c:IT IS DOA! As Collins announced her intentions to vote NO!\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x11\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: IT IS DOA! As Collins announced her intentions to vote NO!\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "28                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               b'\\x80\\x04\\x95\\xb7\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94Cz\\x80\\x04\\x95o\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cILets make some fact based statements. Half of them under 21???? fake news\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X \\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Lets make some fact based statements. Half of them under 21???? fake news\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "33                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  b'\\x80\\x04\\x95\\xd1\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94C\\x87\\x80\\x04\\x95|\\x00\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94\\x8cVthis is *&^%ing outrageous. The prosecutor should be immediately removed from his job.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X-\\x03\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: this is *&^%ing outrageous. The prosecutor should be immediately removed from his job.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "34  b'\\x80\\x04\\x95\\xa9\\x0b\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\xf3\\x03\\x00\\x00\\x80\\x04\\x95\\xe8\\x03\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X\\xbf\\x03\\x00\\x00I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94X\\x96\\x06\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: I believe we have reached any reasonable limit of decency in interfering with the private life of our politicians. Are we really so naive to believe that if the PM would have had the intention of doing a favor to an old friend a free helicopter ride was the thing that had him made it.?  It is pure witch hunt and then we are surprised why the best do not show up for public service. Would any of us have his/her own life thrown apart for a bunch of whiners never happy wanting everything without paying for anything. I am paying probably more taxes than I would like but I am happy and proud to live in a country that is protecting the weakest. This is insane we have completely lost a real perspective. This is not the measurement of the success of a politician, if he takes a free helicopter ride. But unfortunately it is much easier to blame someone else than look at yourself and be honest about our shortcomings. Who is without sin throw the first stone\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  \n",
       "37                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  b'\\x80\\x04\\x95S\\x07\\x00\\x00\\x00\\x00\\x00\\x00\\x8c\\x10autolabel.schema\\x94\\x8c\\rLLMAnnotation\\x94\\x93\\x94)\\x81\\x94}\\x94(\\x8c\\x08__dict__\\x94}\\x94(\\x8c\\x14successfully_labeled\\x94\\x88\\x8c\\x05label\\x94\\x8c\\x05toxic\\x94\\x8c\\x0bcurr_sample\\x94B\\xc8\\x01\\x00\\x00\\x80\\x04\\x95\\xbd\\x01\\x00\\x00\\x00\\x00\\x00\\x00}\\x94(\\x8c\\x07example\\x94X\\x94\\x01\\x00\\x00Don\\xe2\\x80\\x99t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don\\xe2\\x80\\x99t need to. \\n\\nTimmy\\xe2\\x80\\x99s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald\\xe2\\x80\\x99s is moving up their automation timelines.\\x94\\x8c\\x05label\\x94\\x8c\\tnot toxic\\x94u.\\x94\\x8c\\x10confidence_score\\x94N\\x8c\\x0fgeneration_info\\x94N\\x8c\\x0craw_response\\x94h\\t\\x8c\\x0bexplanation\\x94\\x8c\\x00\\x94\\x8c\\x06prompt\\x94Xk\\x04\\x00\\x00Does the provided comment contain \\'toxic\\' language? Say toxic or not.\\n\\nYou will return the answer with just one element: \"the correct label\"\\n\\nSome examples with their output answers are provided below:\\n\\nInput: It\\'s ridiculous that these guys are being called \\'protesters\\'. Being armed is a threat of violence, which makes them terrorists.\\nOutput: toxic\\n\\nInput: This is so cool. It\\'s like, \\'would you want your mother to read this??\\' Really great idea, well done!\\nOutput: not toxic\\n\\nInput: This bitch is nuts. Who would read a book by a woman\\nOutput: toxic\\n\\nInput: It was a great show. Not a combo I\\'d of expected to be good together but it was.\\nOutput: not toxic\\n\\nNow I want you to label the following example:\\nInput: Don\\xe2\\x80\\x99t get it do you. As the price of things go up especially eating out people will respond by not tipping, and layoffs will ensue. I will be dropping my tip rate to 0. After all you are being paid right, so I don\\xe2\\x80\\x99t need to. \\n\\nTimmy\\xe2\\x80\\x99s has already started to raise prices, and I understand from a number of franchises getting ready to cut staff. McDonald\\xe2\\x80\\x99s is moving up their automation timelines.\\nOutput: \\x94\\x8c\\x05error\\x94Nu\\x8c\\x0e__fields_set__\\x94\\x8f\\x94(h\\x08h\\nh\\rh\\x0eh\\x07h\\x0fh\\x11h\\x13\\x90\\x8c\\x1c__private_attribute_values__\\x94}\\x94ub.'  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# review top 10 mistakes\n",
    "display(ds.df[ds.df[\"label\"] !=\n",
    "        ds.df[\"ToxicCommentClassification_label\"]].head(10))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d32d959-4354-4243-bc10-e636132ceb57",
   "metadata": {
    "id": "7d32d959-4354-4243-bc10-e636132ceb57"
   },
   "source": [
    "Based on the mistakes we are making, we will make some adjustments to our `task_guidelines`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a6616a9d-806c-4843-b3cc-4917271a39b9",
   "metadata": {
    "id": "a6616a9d-806c-4843-b3cc-4917271a39b9"
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "        \"delimiter\": \",\",\n",
    "    },\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"gpt-3.5-turbo-0301\",\n",
    "    },\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \\nYour job is to correctly label the provided input example into one of the following categories:\\n{labels}\",\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"few_shot_examples\": [\n",
    "            {\n",
    "                \"example\": \"It's ridiculous that these guys are being called 'protesters'. Being armed is a threat of violence, which makes them terrorists.\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done!\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This bitch is nuts. Who would read a book by a woman\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"It was a great show. Not a combo I'd of expected to be good together but it was.\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "        ],\n",
    "        \"few_shot_selection\": \"fixed\",\n",
    "        \"few_shot_num\": 4,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: {label}\",\n",
    "    },\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3d247d93-8811-43e8-80b0-82343ed0f1a1",
   "metadata": {
    "id": "3d247d93-8811-43e8-80b0-82343ed0f1a1"
   },
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config, cache=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "38f319d8-e48e-4a5e-bcc3-8b57c52dfa6a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 633,
     "referenced_widgets": [
      "5d295fa4575140239b2b41992c27e547",
      "f6e7bcbd235d453394a8dd6ab4fc8366"
     ]
    },
    "id": "38f319d8-e48e-4a5e-bcc3-8b57c52dfa6a",
    "outputId": "4de5442a-f36c-4d67-a0fd-2d94cc356ab9"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66648161740448b8b67fdcce2251f799",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $5.0021 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0025 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$5.0021\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0025\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \n",
      "Your job is to correctly label the provided input example into one of the following categories:\n",
      "toxic\n",
      "not toxic\n",
      "\n",
      "You will return the answer with just one element: \"the correct label\"\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: It's ridiculous that these guys are being called 'protesters'. Being armed is a threat of violence, which makes them terrorists.\n",
      "Output: toxic\n",
      "\n",
      "Input: This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done!\n",
      "Output: not toxic\n",
      "\n",
      "Input: This bitch is nuts. Who would read a book by a woman\n",
      "Output: toxic\n",
      "\n",
      "Input: It was a great show. Not a combo I'd of expected to be good together but it was.\n",
      "Output: not toxic\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(dataset=ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0650f621-c043-45da-bb05-9377e109aa91",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 167,
     "referenced_widgets": [
      "7cf2158a33b54e1b8c2bd142574b5afb",
      "d37d874c5c2e418fb7e988d3ca20a3d3"
     ]
    },
    "id": "0650f621-c043-45da-bb05-9377e109aa91",
    "outputId": "c2c8891c-d528-4fec-848e-0b97ad8de452"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a1583d39433472eb8bd15c401bdb805",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:10 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 88649 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:11 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89034 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:13 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89215 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:15 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89388 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:16 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89626 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-07-11 15:07:17 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89058 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 0.0505\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.77     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└─────────┴──────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.77    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└─────────┴──────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e6a4329c",
   "metadata": {},
   "source": [
    "### Experiment #4: Integrating Chain of thought prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d02638cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_cot = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "        \"explanation_column\": \"explanation\",\n",
    "        \"delimiter\": \",\",\n",
    "    },\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"gpt-3.5-turbo-0301\",\n",
    "    },\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \\nYour job is to correctly label the provided input example into one of the following categories:\\n{labels}\",\n",
    "        \"output_guidelines\": 'Your answer will consist of an explanation, followed by the correct answer. The last line of the response should always be is JSON format with one key: {\"label\": \"the correct answer\"}.\\n',\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"few_shot_examples\": \"data/civil_comments/seed.csv\",\n",
    "        \"few_shot_selection\": \"label_diversity_random\",\n",
    "        \"few_shot_num\": 4,\n",
    "        \"chain_of_thought\": True,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: Let's think step by step.\\n{explanation}\\n{label}\",\n",
    "    },\n",
    "}\n",
    "\n",
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config_cot)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "969be726",
   "metadata": {},
   "outputs": [],
   "source": [
    "seeds_with_explanations = agent.generate_explanations(\"data/civil_comments/seed.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e95b5f25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6005c134b80844318fdaae2740cee6f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $6.2892 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0031 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$6.2892\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0031\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \n",
      "Your job is to correctly label the provided input example into one of the following categories:\n",
      "toxic\n",
      "not toxic\n",
      "\n",
      "Your answer will consist of an explanation, followed by the correct answer. The last line of the response should always be is JSON format with one key: {\"label\": \"the correct answer\"}.\n",
      "\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: Thank you, Roman Kuczynski for your letter, providing insightful information based on true life experiences.  Our current Federal government should pay attention and be far more cautious and protective of Canadians.\n",
      "Output: Let's think step by step.\n",
      "The comment starts by expressing gratitude towards Roman Kuczynski for his letter, which indicates a positive sentiment. The comment then goes on to mention that the current Federal government should pay attention and be more cautious and protective of Canadians. This can be seen as fair criticism of the government's actions or policies, which is allowed according to the task description. The comment does not include any obscenities, racial slurs, sexually explicit material, or insults towards any person, demographic, or race. Therefore, the comment can be labeled as 'not toxic'. So, the answer is not toxic.\n",
      "{\"label\": \"not toxic\"}\n",
      "\n",
      "Input: When all else fails, change the subject to Hillary's emails.\n",
      "Output: Let's think step by step.\n",
      "The comment provided does not contain any obscenities, racial slurs, sexually explicit material, or insults towards a person, demographic, or race. It is a criticism of a political figure, Hillary Clinton, and her emails. According to the task description, criticisms of policy and politicians are marked as 'not toxic' unless they include the aforementioned inappropriate content. Therefore, since this comment does not contain any of those elements, it can be labeled as 'not toxic'. So, the answer is not toxic.\n",
      "{\"label\": \"not toxic\"}\n",
      "\n",
      "Input: Since you've already acknowledged that your primary goal is to be a troll, I'll let this thread die.\n",
      "Output: Let's think step by step.\n",
      "The comment includes an insult towards the person, referring to them as a troll. This violates the rule that comments insulting a person are considered toxic. Therefore, the comment is labeled as toxic. So, the answer is toxic.\n",
      "{\"label\": \"toxic\"}\n",
      "\n",
      "Input: These Liberals are selling their souls cheaply. Still, either for  lousy $1,500.00 or two million, it is still prostitution!\n",
      "Output: Let's think step by step.\n",
      "The comment includes insults towards a specific group of people, in this case, Liberals. It also uses derogatory language by referring to their actions as \"selling their souls cheaply\" and comparing it to prostitution. This comment insults a specific demographic and uses sexually explicit language, which violates the criteria for being labeled as toxic. So, the answer is toxic.\n",
      "{\"label\": \"toxic\"}\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: Let's think step by step.\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6dac624a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "098019d663bc4dcf83c2e5001ed1b3e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 0.1364\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.82     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└─────────┴──────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.82    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└─────────┴──────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds = agent.run(ds, max_items=100)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "BPUpFYVWpZJT",
   "metadata": {
    "id": "BPUpFYVWpZJT"
   },
   "source": [
    "### Experiment #5: Using a different LLM\n",
    "\n",
    "We've iterated a fair bit on prompts, and few-shot examples. Let's evaluate a few different LLMs provided by the library out of the box. For example, we observe that we can boost performance even further by using `text-davinci-003`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "DLJPDWYXphpq",
   "metadata": {
    "id": "DLJPDWYXphpq"
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "        \"explanation_column\": \"explanation\",\n",
    "        \"delimiter\": \",\",\n",
    "    },\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"text-davinci-003\",\n",
    "    },\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \\nYour job is to correctly label the provided input example into one of the following categories:\\n{labels}\",\n",
    "        \"output_guidelines\": 'Your answer will consist of an explanation, followed by the correct answer. The last line of the response should always be is JSON format with one key: {\"label\": \"the correct answer\"}.\\n',\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"few_shot_examples\": \"data/civil_comments/seed.csv\",\n",
    "        \"few_shot_selection\": \"label_diversity_random\",\n",
    "        \"few_shot_num\": 4,\n",
    "        \"chain_of_thought\": True,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: Let's think step by step.\\n{explanation}\\n{label}\",\n",
    "    },\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5qv4WlnXp-0Z",
   "metadata": {
    "id": "5qv4WlnXp-0Z"
   },
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config, cache=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "K0jxOl23qCFo",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 633,
     "referenced_widgets": [
      "14da23e4c0cc4f27a301fb6e27f51210",
      "6ba9a83f387941e99133c9615276a176"
     ]
    },
    "id": "K0jxOl23qCFo",
    "outputId": "eac16072-ea22-435b-dba1-f4911ac3e450"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "659144246d1a47b5937301a19afaad79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬──────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $71.3956 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000     </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0357  </span>│\n",
       "└──────────────────────────┴──────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬──────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$71.3956\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000    \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0357 \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴──────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \n",
      "Your job is to correctly label the provided input example into one of the following categories:\n",
      "toxic\n",
      "not toxic\n",
      "\n",
      "Your answer will consist of an explanation, followed by the correct answer. The last line of the response should always be is JSON format with one key: {\"label\": \"the correct answer\"}.\n",
      "\n",
      "\n",
      "Some examples with their output answers are provided below:\n",
      "\n",
      "Input: Thank you, Roman Kuczynski for your letter, providing insightful information based on true life experiences.  Our current Federal government should pay attention and be far more cautious and protective of Canadians.\n",
      "Output: Let's think step by step.\n",
      "The comment starts by expressing gratitude towards Roman Kuczynski for his letter, which indicates a positive sentiment. The comment then goes on to mention that the current Federal government should pay attention and be more cautious and protective of Canadians. This can be seen as fair criticism of the government's actions or policies, which is allowed according to the task description. The comment does not include any obscenities, racial slurs, sexually explicit material, or insults towards any person, demographic, or race. Therefore, the comment can be labeled as 'not toxic'. So, the answer is not toxic.\n",
      "{\"label\": \"not toxic\"}\n",
      "\n",
      "Input: When all else fails, change the subject to Hillary's emails.\n",
      "Output: Let's think step by step.\n",
      "The comment provided does not contain any obscenities, racial slurs, sexually explicit material, or insults towards a person, demographic, or race. It is a criticism of a political figure, Hillary Clinton, and her emails. According to the task description, criticisms of policy and politicians are marked as 'not toxic' unless they include the aforementioned inappropriate content. Therefore, since this comment does not contain any of those elements, it can be labeled as 'not toxic'. So, the answer is not toxic.\n",
      "{\"label\": \"not toxic\"}\n",
      "\n",
      "Input: Since you've already acknowledged that your primary goal is to be a troll, I'll let this thread die.\n",
      "Output: Let's think step by step.\n",
      "The comment includes an insult towards the person, referring to them as a troll. This violates the rule that comments insulting a person are considered toxic. Therefore, the comment is labeled as toxic. So, the answer is toxic.\n",
      "{\"label\": \"toxic\"}\n",
      "\n",
      "Input: These Liberals are selling their souls cheaply. Still, either for  lousy $1,500.00 or two million, it is still prostitution!\n",
      "Output: Let's think step by step.\n",
      "The comment includes insults towards a specific group of people, in this case, Liberals. It also uses derogatory language by referring to their actions as \"selling their souls cheaply\" and comparing it to prostitution. This comment insults a specific demographic and uses sexually explicit language, which violates the criteria for being labeled as toxic. So, the answer is toxic.\n",
      "{\"label\": \"toxic\"}\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: Let's think step by step.\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(dataset=ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "T0O3-xXwqFMj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 167,
     "referenced_widgets": [
      "16c17440cb4b4074ab057c2b071e2d06",
      "73eabb83a3804a6381ce1b9330fc7466"
     ]
    },
    "id": "T0O3-xXwqFMj",
    "outputId": "773c8b2a-e38f-4924-c0ec-10963d4af8b0"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15cab2949aab4f99b9de3abeded6efa3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 1.7546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.9      </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└─────────┴──────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.9     \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└─────────┴──────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Total number of failures: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Total number of failures: \u001b[1;36m0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b6b16d04-6039-4337-be09-4a87be341159",
   "metadata": {
    "id": "b6b16d04-6039-4337-be09-4a87be341159"
   },
   "source": [
    "### Experiment #6: Using confidence scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "506fd811-734b-4694-87de-cabba81922ed",
   "metadata": {
    "id": "506fd811-734b-4694-87de-cabba81922ed"
   },
   "outputs": [],
   "source": [
    "# Start computing confidence scores (using Refuel's LLMs)\n",
    "os.environ[\"REFUEL_API_KEY\"] = \"sk-xxxxxxxxxxxx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71eab179-d73f-46e7-a539-15e975b0478a",
   "metadata": {
    "id": "71eab179-d73f-46e7-a539-15e975b0478a"
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"task_name\": \"ToxicCommentClassification\",\n",
    "    \"task_type\": \"classification\",\n",
    "    \"dataset\": {\n",
    "        \"label_column\": \"label\",\n",
    "        \"delimiter\": \",\",\n",
    "    },\n",
    "    \"model\": {\n",
    "        \"provider\": \"openai\",\n",
    "        \"name\": \"gpt-3.5-turbo\",\n",
    "        \"compute_confidence\": True,\n",
    "    },\n",
    "    \"prompt\": {\n",
    "        \"task_guidelines\": \"You are an expert at identifying toxic comments. You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material. Any comments that are sexually explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'. \\nYour job is to correctly label the provided input example into one of the following categories:\\n{labels}\",\n",
    "        \"labels\": [\n",
    "            \"toxic\",\n",
    "            \"not toxic\",\n",
    "        ],\n",
    "        \"few_shot_examples\": [\n",
    "            {\n",
    "                \"example\": \"It's ridiculous that these guys are being called 'protesters'. Being armed is a threat of violence, which makes them terrorists.\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This is so cool. It's like, 'would you want your mother to read this??' Really great idea, well done!\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"This bitch is nuts. Who would read a book by a woman\",\n",
    "                \"label\": \"toxic\",\n",
    "            },\n",
    "            {\n",
    "                \"example\": \"It was a great show. Not a combo I'd of expected to be good together but it was.\",\n",
    "                \"label\": \"not toxic\",\n",
    "            },\n",
    "        ],\n",
    "        \"few_shot_selection\": \"fixed\",\n",
    "        \"few_shot_num\": 4,\n",
    "        \"example_template\": \"Input: {example}\\nOutput: {label}\",\n",
    "    },\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd475fe-5fcf-4a5e-a59f-927a77802624",
   "metadata": {
    "id": "7dd475fe-5fcf-4a5e-a59f-927a77802624"
   },
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config, cache=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b09b97c9-ce6e-4f5b-9237-082b29723e7d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 437,
     "referenced_widgets": [
      "ff1aaaae37cf4ac0b060820241fd92ab",
      "1b6586885bbf4f5a88ea2da6c7254946"
     ]
    },
    "id": "b09b97c9-ce6e-4f5b-9237-082b29723e7d",
    "outputId": "7792609c-6344-4bf3-d85d-da2bbc535de5"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ff1aaaae37cf4ac0b060820241fd92ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $4.7442 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 2000    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0024 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$4.7442\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m2000   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0024\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "You are an expert at identifying toxic comments.\n",
      "\n",
      "You aim to act in a fair and balanced manner, where comments that provide fair criticism of something or someone are labelled 'not toxic'. Similarly, criticisms of policy and politicians are marked 'not toxic', unless the comment includes obscenities, racial slurs or sexually explicit material.\n",
      "\n",
      "Any comments that are explicit, obscene, or insults a person, demographic or race are not allowed and labeled 'toxic'.\n",
      "\n",
      "\n",
      "You will return the answer with just one element: \"the correct label\"\n",
      "\n",
      "Now I want you to label the following example:\n",
      "Input: [ Integrity means that you pay your debts.]\n",
      "\n",
      "Does this apply to President Trump too?\n",
      "Output: \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "\n",
    "ds = AutolabelDataset(\"data/civil_comments/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a79c53d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, start_index=0, max_items=100)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "virtualenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0828040fcf104de9935d7a157d225b16": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_43aca6448ea54e8e9ccb8d7a77930812",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n",
         "text/plain": ""
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "14da23e4c0cc4f27a301fb6e27f51210": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_6ba9a83f387941e99133c9615276a176",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Generating Prompts... <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:01</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\n</pre>\n",
         "text/plain": "Generating Prompts... \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:01\u001b[0m \u001b[36m0:00:00\u001b[0m\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "16c17440cb4b4074ab057c2b071e2d06": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_73eabb83a3804a6381ce1b9330fc7466",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:26</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\nCost in $=0.70, support=100, threshold=-inf, accuracy=0.8800, completion_rate=1.0000\n</pre>\n",
         "text/plain": "\u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:26\u001b[0m \u001b[36m0:00:00\u001b[0m\nCost in $=0.70, support=100, threshold=-inf, accuracy=0.8800, completion_rate=1.0000\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "1b6586885bbf4f5a88ea2da6c7254946": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3adc44fde45c472e892a4a0b6a388e87": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "43aca6448ea54e8e9ccb8d7a77930812": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "58b666c148a64a78b0be6c012f3598f1": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_3adc44fde45c472e892a4a0b6a388e87",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:01:01</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\nCost in $=0.04, support=100, threshold=-inf, accuracy=0.6900, completion_rate=1.0000\n</pre>\n",
         "text/plain": "\u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:01:01\u001b[0m \u001b[36m0:00:00\u001b[0m\nCost in $=0.04, support=100, threshold=-inf, accuracy=0.6900, completion_rate=1.0000\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "5d295fa4575140239b2b41992c27e547": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_f6e7bcbd235d453394a8dd6ab4fc8366",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Generating Prompts... <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:00</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\n</pre>\n",
         "text/plain": "Generating Prompts... \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:00\u001b[0m \u001b[36m0:00:00\u001b[0m\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "655ee0e9348c4869a50bc54c2b5237f8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "65f81d6e285e477da1a7ef85be75ad3e": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_655ee0e9348c4869a50bc54c2b5237f8",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:04:10</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\nCost in $=0.04, support=100, threshold=-inf, accuracy=0.7500, completion_rate=1.0000\n</pre>\n",
         "text/plain": "\u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:04:10\u001b[0m \u001b[36m0:00:00\u001b[0m\nCost in $=0.04, support=100, threshold=-inf, accuracy=0.7500, completion_rate=1.0000\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "6ba9a83f387941e99133c9615276a176": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "73eabb83a3804a6381ce1b9330fc7466": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7bdf0bcc39004bbb908b20ab07174d48": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7cf2158a33b54e1b8c2bd142574b5afb": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_d37d874c5c2e418fb7e988d3ca20a3d3",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:01:11</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\nCost in $=0.05, support=100, threshold=-inf, accuracy=0.7700, completion_rate=1.0000\n</pre>\n",
         "text/plain": "\u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:01:11\u001b[0m \u001b[36m0:00:00\u001b[0m\nCost in $=0.05, support=100, threshold=-inf, accuracy=0.7700, completion_rate=1.0000\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "8724335afccc4d8182068498544d0ed1": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_7bdf0bcc39004bbb908b20ab07174d48",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Generating Prompts... <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:00</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\n</pre>\n",
         "text/plain": "Generating Prompts... \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:00\u001b[0m \u001b[36m0:00:00\u001b[0m\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "902b500a2ee741d892791b8d430111b9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d37d874c5c2e418fb7e988d3ca20a3d3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e61c6dd36edf4195a4ac600fc0faa24f": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_902b500a2ee741d892791b8d430111b9",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Generating Prompts... <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:00</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\n</pre>\n",
         "text/plain": "Generating Prompts... \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:00\u001b[0m \u001b[36m0:00:00\u001b[0m\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    },
    "f6e7bcbd235d453394a8dd6ab4fc8366": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ff1aaaae37cf4ac0b060820241fd92ab": {
     "model_module": "@jupyter-widgets/output",
     "model_module_version": "1.0.0",
     "model_name": "OutputModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/output",
      "_model_module_version": "1.0.0",
      "_model_name": "OutputModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/output",
      "_view_module_version": "1.0.0",
      "_view_name": "OutputView",
      "layout": "IPY_MODEL_1b6586885bbf4f5a88ea2da6c7254946",
      "msg_id": "",
      "outputs": [
       {
        "data": {
         "text/html": "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Generating Prompts... <span style=\"color: #729c1f; text-decoration-color: #729c1f\">━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━</span> <span style=\"color: #008000; text-decoration-color: #008000\">100/100</span> <span style=\"color: #808000; text-decoration-color: #808000\">0:00:00</span> <span style=\"color: #008080; text-decoration-color: #008080\">0:00:00</span>\n</pre>\n",
         "text/plain": "Generating Prompts... \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100/100\u001b[0m \u001b[33m0:00:00\u001b[0m \u001b[36m0:00:00\u001b[0m\n"
        },
        "metadata": {},
        "output_type": "display_data"
       }
      ]
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
